Fixes #4691.

Problem

In --durability memory, every append fell back to StreamState::schedule_meta_flush: a per-stream tokio::spawn + 100 ms sleep + spawn_blocking running a full .meta sidecar rewrite (JSON serialize + File::create + rename, plus parent-dir inode contention). Whenever the per-stream inter-append gap is ≥ the 100 ms debounce (any rate ≤ ~10 ops/s/stream — the common case at high stream cardinality), every stream pays a timer task plus a full sidecar rewrite every 100 ms. Wal mode stopped paying this in #4675 (the checkpoint batches sidecar writes), so memory mode — supposedly the cheaper path — burned ~5x wal's CPU at identical load.

Fix

Give memory mode the same batched treatment (packages/durable-streams-rust):

• Store::mark_meta_dirty(st) — appends (memory branch) and TTL read touches now only CAS meta_dirty and, on the false→true edge, push the stream into a store-level meta_sweep set (deduped: at most one entry per stream per cycle). No timer task, no per-append write.
• Store::sweep_meta_once() — drains the set and writes every still-dirty sidecar in one spawn_blocking pass. It skips hard-deleted streams (Arc identity check against the streams map), so a pending flush can no longer resurrect an unlinked sidecar — a race the old debounce had.
• spawn_meta_sweeper (main.rs) — a single 1 s ticker drives the sweep in both durability modes (wal appends still flush via the checkpoint; only TTL read touches use the sweeper there).
• StreamState::schedule_meta_flush is deleted. Durable flush paths (close/delete/shutdown write_meta_sync(st, true) call sites) are unchanged.

The sidecar's producer/access state is documented as a non-durable, lagging flush; the lag bound moves from 100 ms to the 1 s sweep cadence — exactly the trade wal mode made in #4675.

Validation (local ds-bench, kind, server capped at 2 CPU)

ds-bench sustained suite (10 append/s per stream, 256 B payloads, 90 s window, 1 fleet pod), baseline image = main @ 640509c, fix image = this branch:

Memory-mode CPU drops ~4–6x at fixed load and lands below wal at every stream count — the issue's expected outcome (memory does strictly less work per append). Throughput held at the offered rate in every cell (stable=True, zero errors); wal cells are unchanged within noise (its append path doesn't touch this code — only TTL read touches moved to the sweeper).

bench-latency (in-repo host harness, --quick, after the fix): memory-mode write ack mean 0.29–0.51 ms across 1–64 KiB payloads — within the ~0.2–0.5 ms band the issue expects; wal unchanged (2.5–3.3 ms, macOS F_FULLFSYNC dominated).

Tests

• store::meta_sweep_tests::mark_dedupes_and_sweep_flushes — CAS dedupe, sweep persists pending producer state, clears the flag, second sweep is a no-op.
• store::meta_sweep_tests::sweep_skips_hard_deleted_stream — a sweep after a hard delete does not resurrect the sidecar.
• handlers::memory_mode_tests::memory_append_defers_sidecar_to_store_sweep — a memory-mode append no longer flushes the sidecar within the old 100 ms debounce window; the batched sweep is what persists it.
• Full cargo test (105) and the protocol conformance suite (326 tests) pass; clippy clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SmhCex16r9JUPxHunf7Roy